Changes to filename_template checks by abishekg7 · Pull Request #1464 · MPAS-Dev/MPAS-Model

abishekg7 · 2026-06-02T16:55:14Z

Issue:

Previously, if two active, output stream definitions in the streams. file specified the same string as the filename_template attribute, the corresponding MPAS CORE would crash with an error message. This was also the case for when one of the streams, sharing the filename_template, included an inactive package, hence effectively making it inactive. The existing logic to perform stream uniqueness checks in src/framework/xml_stream_parser.c did not have the information about which packages associated with a stream were active, and hence did not account for inactive packages.

Solution

This PR introduces a new subroutine MPAS_stream_mgr_check_filename_template in src/framework/mpas_stream_manager.F to check whether two active streams might potentially be accessing the same file (via filename_template) resulting in file conflicts. More specifically, this subroutine checks that there are no active output streams that share an identical filename_template attribute with any other active input or output streams in the stream manager. An active stream in this context is a stream with attribute active_stream = .true. and is not deactivated by an inactive package and is either an input or output stream with at least one active alarm. This routine can be called from within MPAS_stream_mgr_validate_streams or separately as needed.

This PR also introduces a new function 'MPAS_stream_mgr_get_num_alarms', which returns the number of active alarms for a given stream in the specified direction.

The previous logic to check for unique filename_template attributes in src/framework/xml_stream_parser.c has been removed, while retaining the checks for unique stream names in the same location. Note that the previous logic only check for conflicts between two output streams, whereas the new logic also checks for potential conflicts between an input stream and an output stream accessing the same file.

mgduda · 2026-06-11T20:38:56Z

I've run a few quick tests, and it looks like the core logic is now functioning as expected. I had just a couple of ideas that might clean up the implementation:

The outer loop (with stream1_cursor) could cycle on anything other than an active output stream, and that might obviate the need for checks like (stream1_input .and. stream2_input) in the inner loop.
If the inner loop (with stream2_cursor) iterated only over streams that occur in the stream list after stream1_cursor, that would eliminate duplicate error messages (e.g., "Found identical values ... for ... foobar_1 and foobar_2" and later "Found identical values ... for ... foobar_2 and foobar_1").

abishekg7 · 2026-06-12T05:34:22Z

I've run a few quick tests, and it looks like the core logic is now functioning as expected. I had just a couple of ideas that might clean up the implementation:

The outer loop (with stream1_cursor) could cycle on anything other than an active output stream, and that might obviate the need for checks like (stream1_input .and. stream2_input) in the inner loop.

If the inner loop (with stream2_cursor) iterated only over streams that occur in the stream list after stream1_cursor, that would eliminate duplicate error messages (e.g., "Found identical values ... for ... foobar_1 and foobar_2" and later "Found identical values ... for ... foobar_2 and foobar_1").

I've incorporated the 2nd idea in the most recent commit. But having either the outer loop or the inner loop consider only output streams, appears to miss some filename conflicts, due to the directionality introduced by idea #2.

mgduda · 2026-06-15T19:37:42Z

+                        message = 'Found identical values of the filename_template attribute for multiple active output streams, ' &
+                                  // trim(stream1_cursor % name) // ' and ' // trim(stream2_cursor % name) // &
+                                  ', in streams.<CORE>. This may result in file conflicts.'
+                        call mpas_log_write(message, messageType=MPAS_LOG_ERR)


The error message can be quite long when printed to a single line, e.g.,

ERROR: Found identical values of the filename_template attribute for multiple active output streams, restart and output, in streams.<CORE>. This may result in file conflicts.

It might be better to explicitly write a multi-line error message with multiple calls to mpas_log_write.

@abishekg7 It looks like this suggestion is still pending.

The previous message has actually been modified slightly, and it is split up into two calls to mpas_log_write. Let me know if it needs further changes.

abishekg7 · 2026-06-16T00:48:17Z

I have addressed the remaining suggestions in the last few commits.

mgduda · 2026-07-14T01:19:36Z

@abishekg7 As of now, there are quite a few messages that are written to the log file even when there are no issues with the streams. For example, here's what I get from the log file for the atmosphere core:

 Stream 1 invariant filename invariant.nc  active = T input = F output = F  packages_active T
 Stream 1 input filename x1.40962.init.nc  active = T input = T output = F  packages_active T
 Stream 2 restart filename restart.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: T
 Stream 2 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F pkg_active: F
 Stream 2 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: F
 Stream 2 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 restart filename restart.**-**-**_**.**.**.nc  active = T input = T output = T  packages_active T
 Stream 2 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F pkg_active: F
 Stream 2 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: F
 Stream 2 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F  packages_active F
 Stream 1 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F  packages_active F
 Stream 1 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F  packages_active F
 Stream 1 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F  packages_active F
 Stream 1 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T  packages_active F
 Stream 1 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 surface filename x1.40962.sfc_update.nc  active = T input = F output = F  packages_active T
 Stream 1 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active F

Could you update the PR so that these messages are not written?

abishekg7 · 2026-07-14T23:11:04Z

@abishekg7 As of now, there are quite a few messages that are written to the log file even when there are no issues with the streams. For example, here's what I get from the log file for the atmosphere core:

 Stream 1 invariant filename invariant.nc  active = T input = F output = F  packages_active T
 Stream 1 input filename x1.40962.init.nc  active = T input = T output = F  packages_active T
 Stream 2 restart filename restart.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: T
 Stream 2 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F pkg_active: F
 Stream 2 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: F
 Stream 2 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 restart filename restart.**-**-**_**.**.**.nc  active = T input = T output = T  packages_active T
 Stream 2 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F pkg_active: F
 Stream 2 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F pkg_active: F
 Stream 2 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T pkg_active: F
 Stream 2 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 iau filename x1.40962.AmB.**-**-**_**.**.**.nc  active = T input = T output = F  packages_active F
 Stream 1 lbc_in filename lbc.**-**-**_**.**.**.nc  active = T input = T output = F  packages_active F
 Stream 1 ugwp_oro_data_in filename x1.40962.ugwp_oro_data.nc  active = T input = T output = F  packages_active F
 Stream 1 ugwp_ngw_in filename ugwp_limb_tau.nc  active = T input = T output = F  packages_active F
 Stream 1 da_state filename mpasout.**-**-**_**.**.**.nc  active = T input = T output = T  packages_active F
 Stream 1 output filename history.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active T
 Stream 2 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 diagnostics filename diag.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active T
 Stream 2 surface filename x1.40962.sfc_update.nc  active = T input = F output = F pkg_active: T
 Stream 2 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T pkg_active: F
 Stream 1 surface filename x1.40962.sfc_update.nc  active = T input = F output = F  packages_active T
 Stream 1 diag_ugwp filename diag_ugwp.**-**-**_**.**.**.nc  active = T input = F output = T  packages_active F

Could you update the PR so that these messages are not written?

Done.

abishekg7 added 4 commits June 2, 2026 10:50

first draft

30b74f7

Only crash out if it's an output stream

42399bc

Rename subroutine and call from stream_mgr_validate_streams

74af38b

Remove commented lines

8eb3e56

mgduda requested changes Jun 4, 2026

View reviewed changes

abishekg7 added 4 commits June 4, 2026 15:14

Some cleanups addressing review comments

64e5856

remove trailing whitespace

3bf5404

Negating if conditions, removing unused variables and whitespace fixes

5c7a075

Edits to pass ierr to calling routine

97c6d25

abishekg7 commented Jun 4, 2026

View reviewed changes

Comment thread src/framework/mpas_stream_manager.F

abishekg7 requested a review from mgduda June 5, 2026 15:35

mgduda requested changes Jun 5, 2026

View reviewed changes

Comment thread src/framework/mpas_stream_manager.F Outdated

Comment thread src/framework/mpas_stream_manager.F

Comment thread src/framework/mpas_stream_manager.F Outdated

Comment thread src/framework/mpas_stream_manager.F Outdated

Fixes to address review comments, and cleanup of logging statements

741c495

mgduda requested changes Jun 5, 2026

View reviewed changes

Comment thread src/framework/mpas_stream_manager.F Outdated

Comment thread src/framework/mpas_stream_manager.F Outdated

Comment thread src/framework/mpas_stream_manager.F Outdated

abishekg7 added 2 commits June 10, 2026 09:10

Redoing logic

035a4a7

Using stream_mgr_get_num_alarms in the logic. And some cleanup

f5e7f5c

Modifying logic so inner loop starts with the outer loop iterator

ff95f35

mgduda requested changes Jun 15, 2026

View reviewed changes

abishekg7 added 3 commits June 15, 2026 16:03

Make ierr in get_num_alarms optional and cleanup unused variables

5d54033

Fixing whitespace, comment block and error messages

a30e46c

Reducing number of variables + cleanup

0376fed

Removing temporary logging statement

3e33298

Uh oh!

Conversation

abishekg7 commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue:

Solution

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgduda commented Jun 11, 2026

Uh oh!

abishekg7 commented Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgduda Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

mgduda Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

abishekg7 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

abishekg7 commented Jun 16, 2026

Uh oh!

mgduda commented Jul 14, 2026

Uh oh!

abishekg7 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abishekg7 commented Jun 2, 2026 •

edited

Loading